Infinite RAAM: A Principled Connectionist Basis for Grammatical Competence

This paper presents Infinite RAAM (IRAAM), a new fusion of recurrent neural networks with fractal geometry, allowing us to understand the behavior of these networks as dynamical systems. Our recent work with IRAAMs has shown that they are capable of generating the context-free (non-regular) language anbn for arbitrary values of n. This paper expands upon that work, showing that IRAAMs are capable of generating syntactically ambiguous languages but seem less capable of generating certain context-free constructions that are absent or disfavored in natural languages. Together, these demonstrations support our belief that IRAAMs can provide an explanatorily adequate connectionist model of grammatical competence in natural language. Natural Language Issues In an early and extremely influential paper, Noam Chomsky (1956) showed that natural languages (NL’s) cannot be modeled by a finite-state automaton, because of the existence of center-embedded constructions. A second and equally important observation from this work was that a minimally adequate NL grammar must be ambiguous, assigning more than one structure (interpretation) to some sentences, for example, They are flying planes. The first observation led to the development of Chomsky’s formal hierarchy of languages, based on the computational resources of the machines needed to recognize them. In this hierarchy, Chomsky’s observation about center-embedding is expressed by saying that NL’s are non-regular; i.e., they cannot be generated by a grammar having only rules of the form A ! bC, where A and C are non-terminal symbols and b is a terminal symbol. Whether NL’s are merely non-regular, belonging in the next, context-free (CF) level of the Chomsky hierarchy, or are more powerful, belonging further up in the hierarchy, became the subject of heated debate (Higginbotham 1984; Postal and Langendoen 1984; Shieber 1985). Non-CF phenomena such as reduplication/copying (Culy 1985) and crossed serial dependencies (Bresnan, Kaplan, Peters, and Zaenen 1982) suggested that a more powerful approach, using syntactic transformations (Chomsky 1957) was called for, but some researchers criticized transformations as having arbitrary power and thus failing to constrain the types of languages that could be expressed (Gazdar 1982). Further criticism of the entire formal approach came from observing that even CF grammars (CFGs) had the power to generate structures, such as a sequence followed by its mirror image, that did not seem to occur in NL (Manaster-Ramer 1986), or which placed an extraordinary burden on the human parsing mechanism when they did occur (Bach, Brown, and Marslen-Wilson 1986). Connectionism and Natural Language While debates about the complexity of NL were raging, connectionism was beginning to awaken from a fifteen-year sleep. In connectionist models many researchers found a way of embodying flexibility, graceful degradation, and other non-rigid properties that seem to characterize real cognitive systems like NL. This research culminated the publication of a highly controversial paper by Rumelhart and McClelland (1986) which provided a connectionist account of part of the grammar of English using a feed-forward neural network. The paper was soon criticized by more traditional cognitive scientists (Fodor and Pylyshyn 1988; Pinker and Prince 1988), who cited the non-generative nature of such connectionist models as a fundamental shortcoming of the entire field. Partly in response to these criticisms, many connectionists have spent the past decade investigating network models which support generativity through recurrent (feedback) connections (Lawrence, Giles, and Fong 1998; Rodriguez, Wiles, and Elman 1999; Williams and Zipser 1989). The research we present here is an attempt to contribute to this effort while focusing as strongly as possible on the natural language issues described above. Such an attempt faces a number of challenges. First, despite analysis of how a network’s dynamics contribute to its generativity, it is often uncertain whether the dynamics can support generation of well-formed strings beyond a certain length. That is, it is unknown whether the network has a true “competence” for the language of which it has learned a few exemplars, or is merely capable of generating a finite, and hence regular, subset of the language. 1 Second, it is often easier to model weak, rather than strong generative capacity, by building networks that generate or recognize strings having certain properties, without assigning any syntactic structure to the strings. Third, this lack of syntactic structure inhibits the formulation of an account of syntactic ambiguity in such networks, making them less plausible as models of NL. To be fair, not all connectionists, or cognitive scientists, take seriously the notion that human language has infinite generative capacity. Though we obviously do not have the resources to argue the issue here, we are certain that a model with a provably infinite competence would be more persuasive to the cognitive science community as a whole than would a model without one. In sum, we are concerned with formulating a recurrent network model that rigorously addresses the set of criteria that emerged from the long debate over the complexity of NL. As an candidate,the remainder of this paper presents a new formulation of RAAM (Pollack 1990), a recurrent network model that addresses the NL issues in a principled way.